Estimators of Variance for K-Fold Cross-Validation

1 Motivations In machine learning, the standard measure of accuracy for models is the prediction error (PE), i.e. the expected loss on future examples. We consider here the i.i.d. regression or classification setups, where future examples are assumed to be independently sampled from the distribution that generated the training set. When the data distribution is unknown, PE cannot be computed. Therefore, it is estimated to select a particular model within a learning algorithm and to predict its performance. In experiments aiming at comparing learning algorithms, the expected prediction error EPE (where the expectation is taken over training data sets) may be considered as a more relevant criterion [2]. If the amount of data is large enough, PE can be estimated by the mean error over a hold-out test set. The usual variance estimates for means of independent samples can then be computed to derive error bars on the estimated prediction error, and to assess the statistical significance of differences between models. Note however that these procedures do not aim at estimating the variance w.r.t. estimates of EPE. The hold out technique makes an inefficient use of data which forbids its application to small sample sizes. In this situation, one resorts to computer intensive resampling methods such as cross-validation or bootstrap to estimate EPE. We focus on estimating the variance of EPE, the estimate provided by K-fold cross-validation. While it is known that cross-validation provides an unbiased estimate of EPE, it is also known that its variance may be very large [1]. An accurate estimate of variance should provide means to build confidence intervals on EPE, and to derive statistics to test the significance of observed differences between algorithms .